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SELBERG INTEGRALS, ASKEY WILSON POLYNOMIALS AND 
LOZENGE TILINGS OF A HEXAGON WITH A TRIANGULAR HOLE 

HJALMAR ROSENGREN 


Abstract. We obtain an explicit formula for a certain weighted enumeration of lozenge tilings 
of a hexagon with an arbitrary triangular hole. The complexity of our expression depends 
on the distance from the hole to the center of the hexagon. This proves and generalizes 
conjectures of Ciucu et al., who considered the case of plain enumeration when the triangle 
is located at or very near the center. Our proof uses Askey-Wilson polynomials as a tool to 
relate discrete and continuous Selberg-type integrals. 


1. Introduction 

One of the most influential results of enumerative combinatorics is MacMahon’s 
formula [M] 

H(a)H(b)H(c)H(a + b + c) 

H(a + b)H(a + c)H(b + c ) 

for the number of plane partitions contained in a box of size a x b x c, where 
H(n ) = nL# — 1)!. Equivalently, this identity enumerates lozenge tilings of a 
hexagon with side lengths a, b and c. 

There has been quite a lot of work on lozenge tilings of a hexagon with various 
kinds of holes p] EH EH EEU EEH IHkT IHK21 ICK21IHK41 lET lE2l iHGl EH 
EH EH [OKI El EJ. In the seminal paper [P], Propp conjectured an explicit 
formula for the number of tilings of a hexagon H whose side lengths are almost 
equal, with a small triangle T removed from the center of H (more precisely, in 
the notation explained in 1 12.11 this is the region H \ T with a = b = c, m = 1, 
M = N = 0). This conjecture was proved in |Cli [HG ]. More generally, Ciucu et 
al. [ CEKZ] enumerated the tilings when the side lengths of H and T are arbitrary, 
but T is still positioned at (or very near) the center of H. They also conjectured 
enumerations for some adjacent positions of T. 

In the present paper, we consider the general case, when the position of T within 
H is arbitrary. Our main result, Theorem 12.11 expresses a weighted extension of 
the number of tilings as a determinant, whose complexity depends on the distance 
of T from the center of H. Thus, it is a closed formed evaluation if the position of 
T relative to the center of H is fixed, but the side lengths of T and H are arbitrary. 

As in [ CEKZj . the starting point of our proof is the Gessel-Viennot method 
Eg, which gives an explicit determinant formula for the weighted enumeration. 
In [ CEKZ ] the determinant is computed using the method of identification of 
factors |K2j . It seems very difficult to handle the more general determinants that 
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we encounter in this way. Instead, we derive a chain of intermediate expressions 
for our weighted enumeration as indicated in the following diagram. 


t t t . i , -1 . . Gessel-Viennot 

Weighted enumeration - > 

40 


Determinant I 

m 


minor expansion 


Determinant II 

433] 


Cauchy-Binet , n n • i 

< - - - Discrete Selberg integral 


433] 


Christoffel-Heine 


^ . n ii . , n Christoffel-Heine 

Continuous beiberg integral - -—- - > 

433] 


Determinant III 


Here, Determinant I is obtained by the Gessel-Viennot method. It has com¬ 
pletely factored entries and its dimension is equal to one of the side lengths of H. 
Applying an appropriate minor expansion leads to a multivariable basic hypergeo- 
metric series. As it contains the factor EL< •,(//”' —Q mi ) 2 , with mj being summation 
indices, it can be considered as a discrete Selberg-type integral [FWj. Special cases 
of this sum appear in [CEKZ], but are considered there as consequences of the 
enumeration rather than as a tool. 

In general, Selberg-type refers to hypergeometric series or integrals containing 
factors like n*<y I x j ~ x i | c , where the archetypal example is the integral 

P n 

/ n - 2y) 6-1 dxj. 

■H 0 ’ 1 ]" l<j<j<n j =1 

The cases c = 1 and c = 2 are determinantal, in the sense that they can be 
expressed as determinants of one-variable integrals. In our setting, an application 
of the Cauchy-Binet identity leads to an alternative determinant formula for the 
weighted enumeration, Determinant II. It is quite different from Determinant I as 
its entries are Askey-Wilson polynomials and its dimension is equal to the side 
length of T. Using classical results on orthogonal polynomials due to Christoffel 
and Heine, we can rewrite Determinant II as a continuous Selberg integral, where 
II i<j( x j~ x i) 2 i s integrated against the Askey-Wilson orthogonality measure. The 
key observation is now that the results of Christoffel and Heine can be applied 
in a different way to the same Selberg integral. This leads to our end result, 
Determinant III. Here, the matrix entries are again Askey-Wilson polynomials, 
but in base q 2 rather than q. The size of the determinant is related to the distance 
from T to the center of H. 

The above decription of our proof is not quite accurate, as we glossed over 
two important technical aspects. First, Determinant I only applies when the side 
length m of T is even. To extend our result to odd m, we need an a priori 
result on how our weighted enumeration behaves as a function of m, Lemma 13.51 
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Figure 1. The region H\T described by (a, b , c, to, A, B , C) = (2, 3, 2, 2,4, 2,1). 
A tiling of this region, with each horizontal lozenge labelled by its height. 


This is achieved by another application of the Gessel-Viennot method. Second, 
the orthogonality relation for Askey-Wilson polynomials is actually not valid for 
the specific parameters appearing from the tiling problem. Thus, the continuous 
Selberg integral mentioned above does not make sense as an expression for the 
weighted enumeration, but only appears after continuation to a different range of 
parameters. 

The explicit expression given in Theorem 12.11 is admittedly rather complicated, 
but we believe that the method of proof is more interesting than the result. It 
seems likely that there are other problems related to tilings and plane partitions 
that can be approached with similar methods. For instance, one could ask for a 
“dual” of our result in the sense of [CK2j . 

Acknowledgements: This research was partially supported by the Swedish 
Science Research Council. All figures are based on packages written by Theresia 
Eisenkolbl and Christian Krattenthaler. 

2. Main result 

2.1. Weighted enumeration of tilings. Consider the triangular lattice in the 
plane, formed by equilateral triangles of side length 1 and height (j) = \/3/2. 
On this lattice, we draw a convex hexagon H and remove an equilateral triangle 
T C H . We are interested in tilings of H\T by lozenges, that is, by quadrilaterals 
formed by adjoining two adjacent lattice triangles. Using the bijection to lattice 
paths discussed in §3.11 it is easy to see that, for such tilings to exist, H must have 
consecutive side-lengths a, b + m , c, a + m, b, c + m, where m is the side-length 
of T. Moreover, the sides of T must be parallel to the long sides (of length a + m, 
b + m and c + m) of H. 

We will refer to the sides of H by the expression for their length; for instance, 
the side b + m is the second side in the ordering given above. We will picture the 
region H\T as in Figured This allows us to use terminology such as “horizontal” 
to refer to the direction orthogonal to the side c. 
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To specify the position of T within H , let A(f>, B<p and C(f> denote the distance 
from T to the line containing the side a + m, b + m and c + m, respectively. It is 
easy to see that 

A + B + C = a + b + c, (2.1) 

for instance, by applying Viviani’s theorem to the triangle formed by extending 
the short sides of H. The distances from T to the lines containing the short sides 
of H are [b + c — A)(j), (a + c — B)(j) and (a + b — C)4>. Thus, 

0 < A < 6 + c, 0 < < a + c, 0 < C < a + b. (2.2) 

Conversely, any non-negative integers a, b, c, A, B , C and m subject to (12. lj) 
and (12.2ji describe a region H \ T. Note that we include degenerate cases when 
some sides of H have length zero, when no triangle is removed (m = 0) or when 
T touches the boundary of H. We will also specify the location of T by the 
coordinates 

M = 2A - b - c, N = 2B - a - c, (2.3) 

so that 

. b + c + M a + c + N a + b — M — N 

A= -, B = -, C= -. 

2 2 2 

Then, M and N are integers of the same parity as b + c and a + c, respectively, 

such that 

\M\ < b + c, |IV| < a + c, \M + N\<a + b. 

As an example, the region in Figured] corresponds to ( M,N ) = (3,0). 

Note that (M, N) = (0, 0) corresponds to T being located at the center of H. 
This case, and a few other cases with T nearly central, were studied in jCEKZj . 
To be precise, these authors enumerated the tilings when (M, N) equals (0, 0) and 
(0,1) and conjectured enumerations when ( M,N ) equals (0,2) and (0,3). In the 
present paper, we will explain how to prove these conjectures and obtain analogous 
results for any M and N. 

More generally, we will consider a weighted enumeration of tilings. We define 
the height h of a horizontal tile Q to be the vertical distance from the center of 
Q to the center of T (see Figured])- Our main object of study is the partition 
function 

E II p-4) 

tilings of H\T horizontal tiles 

In particular, Z( 1) is the total number of tilings. Our weight function is a special 
case of weights introduce in [BOR] for plane partitions and [S2] for lattice paths. 

Note that, when m — 0, Z(q ) is different from the volume generating function 
for plane partitions computed by MacMahon. Up to a power of q, the latter is 
equal to 

zw = e n u, 

tilings horizontal tiles 
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where h is the vertical distance from the center of a tile to the bottom corner of H. 
When m > 0, the function Z behaves better than Z, being given by completely 
factored expressions in situations (e.g. (. M,N) = (0,0)) when such expressions 
exist for the enumeration problem. 


2.2. Notation. We will write sgn(k) = 1 for k > 0 and sgn (k) = —1 for k < 0. 
Recall also the standard notation ES 

71—1 

(a; q)n = nt 1 _ 

j =o 

(&1, ••• ? ^777,7 9)71 (^lj 9)77 ’ ’ * (^7777 9)775 
(Xl, ... 5 ^r +1 


r+lyr 


bu...,b r 


;q,x = 


Z 

k =0 


(® 1 ) • ■ ■ ) a r+ 1 ] q)k _^k 

(q,bi,. . . ,b r ;q) k 


When x = (aq,... ,x m ), we will write A(x) = — Xi) and also use 

notation such as A (q k ) = W 1 < i< j< m {q kj — g fei )- We introduce the multiple basic 
hypergeometric series 

(ai,.. .,a r+1 -,q) k .x k * 


T’+IS 


ri m) ( a Z'’X 1 ’ q ' x i = id A(y) 2 n 


0<ki<k2<---<k 

n 


L = i (q,b\,..., b r ] q)b 


(2.5) 

In view of the factor A (q k ) 2 , it can be thought of as a discrete Selberg-type integral. 
We introduce the g-hyperfactorial 


H q {m) = 


n m / — A d 

3=1 yi 2 - a 

n m +l ( i — i 
7=1 9 4 2 


L \ m -3 


i_ 1 

q2 4 


m+~-j 


m = 0,1,2,..., 

, m = - 1/2,1/2, 3/2,.... 


( 2 . 6 ) 


Equivalently, 

H q (m) = 


1 i( " +I) n”,(«;«)j-i. 

r T m +h / I 
D= 


m = 0,1, 2,..., 

g -^r 3 +i ) m = -1/2,1/2,3/2, 

Deleting the prefactor from these expressions, we will write 


H q (m ) = 


11^1(9; 9 )j-i, m = 0 , 1 , 2 ,..., 

™ = -1/2,1/2,3/2, 


We use both notations since our main result is easier to state in terms of H q , but 
for the proof it is often more convenient to work with H q . 


We will sometimes write H + = H q and 


H A m ) 
H < {m) - im 


n m / —A . 2 

7 =i (9 2 + q < 


i \ m ~ 3 


n m+2 / , 

7=1 U 4 2 +9 


1 _ J 


i _ 1 


m+^-j 


m = 0 , 1 , 2 ,..., 

, m = -1/2,1/2, 3/2,... 
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Repeated arguments stands for a product; for instance, 

Hq (dl, . . . , fl ra ) Hq (fll ) Hq ((l m ). 

Similar notation will be used for H q . We collect some useful facts about the 
functions II q and H q in an Appendix. 

2.3. Statement of main result. Our main result is formulated in terms of de- 
tGrmiii&iits 

Q MA, ”(a,/3,7;«) = , det S, r g)) , (2.7) 

1 <j,k<M+N J 

labelled by non-negative integers M, N, n and generic parameters a, (3, 7 . The 
matrix entries Qjl Nn (a, /3, 7 ; q) are given for for 1 < k < M and n + j odd by 


(a 2 , a 2 7 2 ; q 2 ) {n+j 1)/2 (a 2 /3 2 ] g 2 ) fc i 


q\( n +j- l )(. n +j-S)+( k 2 1 ) a n+j+k-2pk-l^\{n+j-l) 

X 403 

and for 1 < k < M and n + j even by 

(<*V, «V; g 2 ) (n+j 2 )/ 2 (a 2 0 2 ; g 2 ) fc 1 


q 1 j n ,a. 2 f3 2 ^ 2 q n+j 3 ,a 4 q 2k 2 ,q 2 2k _ 2 2 
a 2 , a 2 /3 2 , a 2 y 2 ’ ^ ^ 


(ag 


fc-i 


a 4 g* k 


K 


g lb+j- 2 ) 2 +(V) a n+i+J ; -3^-l 7 5(n+i-2) 


X 4 <; 


g 2 J n , a 2 /3 2 'y 2 q n+j 2 , a 4 g 2fc 2 , g 2 2k 


2 2 

;g ,9 


( 2 . 8 a) 


( 2 . 8 b) 


a 2 g 2 , a 2 /3 2 , o 2 y 2 

For the remaining cases (M + 1 < k < M + N ), they are determined by 

0, 7 ; ?) = Q,T B 09,«, 7 ; g), i<k<N. 

Though the structure of this determinant may seem complicated, we will see in 
1 13.51 that it appears naturally in the context of Askey-Wilson polynomials. It is 
easy to check that 

Q MNn (a, 13, 7 ; q) = (- 1 ) M N+n(M+N) qMNu (q,- 1 ^ ^- 1 ^- 1 . q - i). ( 2 .9) 


The matrix elements (12.81) are Laurent polynomials in a, (3 and 7. In particular, 
we may (and will) specialize these variables to points where the 403 -sums without 
the prefactor are singular. Note also that each matrix entry is a sum of at most 
max(M, N ) terms. Thus, the following result gives a closed form evaluation of 
Z(q) for fixed M and N. 


Theorem 2.1. With £ = sgn(MN), we have 

Z(q) = CQ IMUNI ’ b (q^ (1 - b - c - m - IMI \-£q^ 1+a+c+m -\ NI) ,q^ m+1) ;q), (2.10) 

where 

(_l)(^) £ ('f )+ N ib+M) 

2 \ m(a-\-b+M+N)-\-ab— ^5^+ f max(|a—6|, | M—N\) H q (\M\, |iV|) 
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x H q 2 


X H q 2 


X 



'a' 


(I ~h 1 


'a' 

+ 

m + 1 

(I ~h 1 

l 

.2. 

5 

2 

5 

.2. 

2 ’ 

2 


"6' 


'6 + 1' 


'6' 

+ 

m + 1 

"6+1" 

l 

_2_ 

J 

2 

5 

_2_ 

2 ’ 

2 


+ 

+ 


m — 1 
2 

m — 1 


}J 2 j~ c+|A^|+l j |- c+JM| j rn+l j~ e+|M|+l j m.-l 


tt (a+b-M-N a+b+M+N 

- n <? V 2 ’ 2 


X 


+ m) H q 

^’77+ 


a+b—\M\ — \N\+m+l q+fr+|Ar| + |JV|+m-l 


+ m) 


2 [ a-\-c—\N\-\-m—l \ ^ ( a-\-c—\N\ 


X 


l r y 2 > 2 i 

Hq(a + c + m) 

f a+c— |7V|+m a+c+|IV|+m—1 a+c+|7V|+m 


+ H q 4, ^+c-|JV|+m+l ^ 


X 


TT ( b+c—M \ 

-''V V 2 ) 


X 


T ( b+c—M , fe+c+|A-/|+m+l \ tt ( b+c+\M\ 

L q 2 y 2 ' > 2 y 11 q ^ 2 

H~(b + c + m ) 


b+c+|Af|+m—1 
l 9“ V 2 




— 6+c—|Af|+m b+c— |M|+m+l b+c+IMI+m^ ( 

? 2 ^ 2 > 2 > 2 j 9 V 


_ (I a—b+M—N I I a-b-M+N 


) 


X H q 2 


X H q 2 


a + b + c — \N\ 
2 

a + 6 + c — |1V| 


+ m, 


a + 6 + c— |1V| + 1 


+ m 


+ 


m + 1 


a + b + c — |iV| + 1 


+ 


m — 1 


Hz 


—e ( a+fr+2c—|M| + |-/V| ■ Q,H-^+2 cH~| M | — |AT| 

2 ' ,n ' 2 


X 


+ m 


[J—e f a+fr+2c— |M| — |iV| _|_ ^ a+6+2c+|M| + |./V| _|_ ^ 
9 V 2 2 


( 2 . 11 ) 


As an example, consider the case (M, IV) = (2, 0). Assuming also that n is odd, 
we have 

|Qii Q 12 I 


g 2 ’°’>,/3, 7 ;g) = 


^ 2 i g 22 


where 


n _ ( _ _i, (a 2 g 2 ,a 2 7 2 ;g 2 ) (n -i)/2 

5l1 ' " ^ g |(n-l) 2 a n~l 7 i(«-l) ’ 

^ , _1 _i,(« 2 9 2 ,« 2 7 2 ;? 2 )(n-i)/2(l -a 2 /? 2 ) 

gi 2 = (ao — a q )-n---r-— - 

qz^- 1 ) cpiy+ph 1-1 ) 

(1 - ^-"Xl - aWg"- 1 )!! - aV)(l - g~ 2 ) 2 
1 (l-g 2 )(l-a 2 g 2 )(l-a 2 /9 2 )(l-a 2 7 2 ) 
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Q21 


Q 22 


(a 2 ,«V;g 2 ) (n+1)/2 

q\( n + l )( n ~ 1 ) a n+l^{n+l) ’ 

(a 2 , « 2 7 2 ; g 2 ) (n+ i)/ 2 (l — a 2 ft 2 ) 

gf4'( ri "^l)C ri 1) q/H+2 l^ry 2 

w A . (l-^-^l-ayTVKl^V)^-?- 2 ) 2 

v (1 — g 2 )(l — o 2 )(l — oi 2 (3 2 )(l — a 2 7 2 ) g 


This can be simplified to 

^2,o,ru„. « _„n_ ( 1 ’?)(« 2 ;9 2 )(n+i)/2(aV,a 2 7 2 ,a 2 7 2 g 2 ;g 2 ) (n _i)/2 

V Pi 'll Q) — /n\ . . 

cp 2 ' + 1 a 2 n+ 2 /37 n 

x {(1 + a 2 g)(l - a 2 /3 2 )(l - a 2 7 2 ) - (1 + q~ n )( 1 - - «V)}. 

If (a,/3, 7 ) = (q x / 2 ,±q y / 2 ,q z / 2 ), the leading Taylor coefficient of this function at 
q = 1 is a completely factored expression times 

(x + y)(x + z) — 2(x + l)(:r + y + z + n — 1 ) 

Substituting (x , y, z, n) 1 —> (— b — c — m — 1, a + c + m + 1, m + 1, b) we find that, 
if a, b and c are all odd and ( M,N ) = (2,0), then Z(l) is a completely factored 
expression times 

(b — a)(b + c) + 2{b + c + m)(a + m) = (a + b){b + c) + 2 m(a + b + c + m). 

After interchanging a and b, we recover the second half of C'KKX , Conj. 1], In 
this way, jCEKZl Conj. 1 and Conj. 2] can both be obtained as special cases of 
Theorem 12.11 

An intriguing consequence of Theorem 12.11 is that Z ( q ) is invariant under the 
transformation (M, N ) t—)■ (— M, —N), up to an elementary prefactor. This means 
that the position of T is reflected in the center of H , see Figure [2] It would be 
interesting to have a conceptual explanation for this unexpected symmetry. 

Corollary 2.2. Denoting by Z MN the partition function Z(q) with fixed values of 
a, b, c, m and q, we have 

ry 1 TJ (a-\-b-\-M-\-N a+b—M—N \ 

£MN _ -L n q\ 2 * 2 r HI) 

Z_ M ,_ N ~ 2 m l M+7V ) H q {sdkfzzlL, a+b+M+N + 

^ H q 2 (2±|dV, a±c±N + mj + 

X H q 2 (2±f^, 2 +^A + fr±£±M, If+cffiM + m J- 

As an example, removing the left triangle in Figure [ 2 ] corresponds to 

(a, b, c, m, M, N) = (2, 5, 2,1,1, 2), 
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Figure 2. Removing one of the two indicated triangles leads to partition func¬ 
tions related by an elementary multiplier. 


which gives 


Z(q) 


(1 + g) 4 (l + g 2 ) 5 (l + g 3 ) 3 (l + g 4 ) 5 (l + q 5 ) 

2 13 g 28 


/(<?), 


with 

f(q) = q S + q‘ + 2 q 6 + 3 q 5 + 3g 4 + 3 q 3 + 2q 2 + q + 1. 
For the right triangle, corresponding to (a, b, c, m, M, N ) = (2, 5, 2,1, 


- 1 , 


- 2 ), 


Z(q) 


(l + g) 3 (l + g 2 ) 4 (l + g 3 ) 2 (l + g 4 ) 4 (l 
2 4 V 5 (1 - q) 



/(<?)• 


In particular, substituting q — 1 we hnd that there are 544 = 2 5 • 17 tilings in 
the first case and 1360 = 2 4 • 5 • 17 in the second case, where the symmetry is 
responsible for the relatively large common prime factor /(1) = 17. 


3. Proof of Theorem 12.11 


3.1. Lattice paths. Following [CEKZj, we study the partition function Z(q) by 
applying a bijection from lozenge tilings to families of non-intersecting paths in 
the square lattice. Given a tiling of H\T, we mark the midpoints of the edges on 
the side b+m and construct paths ending at these points by following the direction 
of the lozenges. This gives m paths starting at the adjacent side of T and b paths 
starting at the side b. We then apply an affine transformation mapping the steps 
of the paths to edges in the square lattice. 

More precisely, with the conventions illustrated in Figure [3l tilings of H\T are 
in bijection with families of up-right non-interesecting paths starting at the points 
(-Pj)jtj' 1 and ending at the points (Qj)j=T\ where 


Pi 

Qj 


I (j-M-j), 1 <j<b, 

1 (C — b + j — 1, A + b + m — j), b + l < j <b + m, 

(a + j — l,b + c + m — j), 1 < j < b + m. 


(3.1a) 

(3.1b) 
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Figure 3. Lattice paths corresponding to the tiling in Figure [1] The circled 
points, denoted Rj, will appear in t)3.31 


In this setting, the weight function becomes a weight on horizontal steps, given by 

(x+2y-Z)/2 , f-x-2y+Z)/2 

- -- (3.2) 

for the step from (x — 1 ,y) to (x,y), where 

Z = 2 A + C + m — 1. 

We now recall the Gessel-Viennot method for weighted enumeration of lattice 
paths [GV] . Consider, in general, an arbitrary weight assigned to each horizontal 
edge in the square lattice. We define the weight of a family of paths to be the 
product of the weights of all horizontal steps in the family. Let 

W ( P \, . . . , P n , Qh ■ • • , Qn) 

denote the sum of the weight of all families of n non-iutersecting up-right lattice 
paths, where the i:th path starts at Pj and ends at Qi, for 1 < i < n. We then 
have the following fundamental result of Lindstrom [Li], see |K3] for a historical 
discussion. 

Lemma 3.1 (Lindstrom). The following identity holds: 

det (w(Pj] Q k )) = ^ sgn(cr )w(Pi, ..., P n ; Q a m,..., Q a (n))- (3.3) 

Ki,k<n z —' 

aeSn 

In other words, in the Laplace expansion of the left-hand side, the contributions 
from intersecting lattice paths cancel. Consider now the case when the points are 
given by (13.ip . Since the paths starting at T will end at consecutive points on the 
side 6 + m, not all permutations will contribute to the sum in (13.3p . In particular, 
if m is even, only even permutations contribute, which means that the right-hand 
side of (13.3(1 reduces to our partition function. 
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Corollary 3.2. If the side length m of the triangle T is even, the partition function 
02.41) has the determinant representation 

z ( q ) = , det {w{PpQ k )). (3.4) 

If m is odd, we get instead a determinant representation for a sign-variation 
of the partition function, generalizing the (—1)-enumeration studied in j lCEKZ ]. 
Although our methods can be adapted to study this function, it will not be con¬ 
sidered in the present work. 


3.2. The partition function as a discrete Selberg integral. Continuing 
in the footsteps of Ciucu et al. [CEKZ] . we rewrite the determinant of Corol¬ 
lary [X21 as a discrete Selberg integral. The following result is a special case of [32] 
Prop. 2.1 (c)]. 

Lemma 3.3 (Schlosser). One has the determinant evaluation 
det (w((xi + j - 1, j/i + m - j)\ (x 2 + 4 , 2/2 - 4))) 

l<j,k<m 

x T~T (g; Q)x 2 +y 2 -x 1 -y 1 -m+j(-q Z ~ X2 ~ yi ~ y2 ~ m+j ] q) Xa - Xl - j+ i+ h 

j = l {Qi Q)x 2—xi+lj (Qi 0)y2—yi—lj 

where |Z| = Y^j=ih an< ^ 

3 (m\ 1 (. 

2V3 ) + 2[ 2 ) i ~ 3Xl + 2X2 ~ 2yi + Z ~ 1) 

+ |m(i 2 - xi)(xi + x 2 + 4yi — 2Z + 1) 

1 ,"4 // \ 1 

+ 2 ( 2 ) + 2 + ^ yi ~ Z + 

3 =1 ' ' 


The exponent X looks complicated, but can be determined from the fact that the 
weight is invariant under q 1//2 g~E 2 . If we would use a more symmetric notation 
(based on q -numbers g -a / 2 — q a / 2 rather than 1 — q a ), the resulting expression would 
be simpler. 

Let us now expand the determinant in Corollary 13.21 into minors according to 


det (w(Pj]Q k )) = 

1 <j,k<b+m V J ' 


)+6ra 

x Qi k + 1)) .det (^(Pft+j; Q tfc+1 )), 


where 4 < • ■ ■ < lb is the ordered complement of {4, ■.., l m } in [0, b + m — 1]. 
The determinants on the right-hand side may be evaluated using Lemma 13.31 
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Rewriting all factors involving the indices L in terms of L, using in particular 


A (q 1 ) = )KI+Er=! Ci) 


n£r(?; ?),--! 


n',=i(?; 9 k-(9; gWm-i-L- 


A(g z ) 


gives 




^ ^*)) = 2 m(a + b-0 +ab 

(q\q) 


l<j,k<b-\-m 

b 


b-\-m 

n 

3=1 


(95 q)j-i(~q 


A-B-b+l. 


1 q)a+j — 1 


(?; q)a+j-i(q-,q)c+j-i 


X 


X 


1 a+c+m+j- 


-q 


A-B-b+l. 


; ?)j- 


1- ^(_l)|f|+^+(?) g Er=i (1)+KI^- 6 —+2) A (g') S 


n 

i=i 

m 

n (g; g) z .(g; g) a -c+i,-(9; g) b+rn _i_ z .(g; q) b+c+m _ A -i-i j {-q A ~ B ^ b+l \q) 


(9; 9 )b+j-i( 9 ; 9)0+1,-(9; q)b+c+ m -i-i j (-q 




5 — C —j+l+Zj 


fl-J-Zo 


where the sum is over indices satisfying 

max(0, C — a) < l\ <■■■< l m < min(6 + m — 1, 6 + c + m — A — 1) (3.5) 

and where 


1 X TT) \ 1 

X = -(£ — A — c) ( 2 j + -((a - C)(a - C + 1) + 6(36 - 4A - 2C - 1)) 


m 


H— ab(a + 36 — 4A — 2C). 

4 

In the notation (12.31) and (12.5ft . this identity can be expressed as follows, where 
we have rewritten the prefactor in a way that will be convenient later. 

Proposition 3.4. 




q 


nim(a+b+M+N)+ab J- J. 
Z 3=1 


(q; q) a+ c+m+j-i(q; q)j-i(-q l2( a b+M N)+J ;q) c 

(q;q) a+j—l (9; q) c-\-m+j— 1 


X 


n 

3 =1 


^^1— b —c— Hi H-^(a — b-\-M-\-N) ^1— b-\-M —AT). ^ 

g ^_ i( ' g 2-2m-b-c+M. ^2)^ ( g a+l. g)^.^ 

(9 2 ;9 2 )l( a +c+lV)+j-l 


X 4 (f). 


( 9 ; 9 )ii+MAi±Jv +i _ 1 (g 2 ; g 2 ) i)+ c-M + ._ 1 

( m ) I q 1 ~ b ~ m 7 q a+1 , + 

^l-b-c-m^l+§-| + 4f+ -f^ —gl+f-f + T'-T ’ P ^ I ’ 


(3.6) 
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where 

X = -2^j - l( a -3b + M + N + 4)(^j - i ab(2c + M-N ) 

— b(a — b) + -^(a — 3 b + M + N)(a + b + 4c — 3 M + N + 2)^ m, 

Note that the factor nr- h~M+N) ; q'j b+ -_ x may vanish. If this is the 

case, the 4 <p^ 11 in (13. 6 p should be interpreted as a sum over indices lj such that 
this zero is cancelled by njli (^q 1+ 2 ^ a ~ b+M+N -F q)^ 1 . This gives the restriction 
1 1 > (—a + b — M — N)/2 = C — a as in (13.511 . 

3.3. The partition function as a function of m. Corollary 13.21 is only valid for 

even values of m. In order to study the case of odd m, we will need the following 
fact. An analogous result for the enumeration Z( 1) was proved in [ CEKZ1 §6]. The 
simple proof given there seems difficult to extend to the weighted enumeration, so 
we use a slightly different approach. 


Lemma 3.5. For fixed values of a, b, c, A, B and C, 

j j \ a-\-b—C 

q2 + q -2 


Z(q) = n 

3 = 1 

where f is a rational function independent of m. 


f (vV . 


(3.7) 


Proof. Given a lozenge tiling of H \ T, we split F[ in two parts, separated by the 
line L containing the side of T parallel to the side b. Recall the bijection to lattice 
paths described in 1 )3.11 The paths starting at the side b cross L at b line-segments, 
which are marked with circles in the left part of Figure [3] Ordering them from left 
to right, let Xj denote the distance from the j:th segment to T. If l is the number 
of crossings to the left of T, the numbers Xj are restricted by 

0 < xi < ■ ■ ■ < xi < min(C — 1, b + c — A — 1), (3.8a) 

0 < xi .|-i < • • • < Xb < min(A — l,a + b — C — 1). (3.8b) 

Applying the same affine map as in 1 )3.11 the circled points are mapped to 

_ f (C - Xj — 1, A + Xj + m), 1 < j < l, 

[ (C + Xj + m, A — Xj — 1), l + 1 < j < b. 


The lattice paths split into a family of b paths starting at (Pj) b j =l and ending at 
(Rj) b j =1 and a family of b + m paths starting at (Sj^ff and ending at (QjffLfb 
where 

{Su ■ ■ ■ j Sb+m) (^1) ■ ■ • > Rli Pb+1, • • • 5 Pb+mi Pl-\- 1, ■ ■ ■ - Pb) • 

More explicitly, Sj = (C + yj, A + m «- yj — 1), where 
(2/1, • • •, Vb+m) = (-zi - 1, • • •, -xi - 1, 0,1,..., m - 1, x t+1 +m,...,x b + m). 
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Applying Lemma 13.11 to both families, only the identity permutation contributes 
to (13.31) . and we find that 

z (o) = R k)) det (w(Sj-Q k )), (3.9) 

z ' 1<?,A '<b 1 <j,k<b-\-m 

where the sum is over all solutions to (13. 8p . for 0 < l < b. In contrast to (13.41) . 
(13.91) holds regardless of the parity of m. 

The first determinant in (13.91) can be computed using Lemma 13.31 By the 
symmetry 

wz{{a, b)) ( c , d)) = w 1 -z((~c, -d); (-a, -b )), 

where we indicate the Z-dependence in (j3.2[) by a subscript, the second determi¬ 
nant can be expressed as 

(_l)( + 2 ) det (wx-zi-a-b-m+j, 1-c-j; -C-y k , l-m-A+y k )), (3.10) 

l<j,k<b+m 

which is again computed by Lemma [3.31 

It will be convenient to write / ~ g if, as a function of m, f /g = h(q m ^ 2 ), with 
h rational. We need to prove that 

Z(q) ~ W a+h ~ c , (3.11) 

where 

m j_ _j_ 

3 = 1 

Consider first (13. 101) . which is obtained by substituting 


(m,x 1 ,x 2 ,yi,y 2 ,lk,Z) 

i —y (h T Try 1 — a — b — m, —C, 1 — b —■ c — m, 1 — m — A, —y k , 2 — 2 A — C — m) 


in Lemma 1X51 Under this substitution, 2( 2 ) m ^ 2 \ l \ ^ 2 m ( c a ^ and 

X o (\ | 4b— a+C — 31+2 ( 

Q rs -' ^ \3/ T 2 V 2 / # 

The factor A (q l ) H> ni<j<fc<&+m(9 _S,fc — Q~ Vj ) splits into six parts, depending on 
whether j and k belong to the interval [1, /], [l + 1, l + m] or [l + m + 1, b + m]. 
The three parts with neither j nor k in the middle interval are clearly rational in 
q m . This leaves us with 


n (i 1 

1 <7<Z, 1 <k<m 


-* -ci x ‘ +i ) 


ii n - 4 i_j ) 


where the first factor can be written 

i 


l<j<m, l</c<6—/ 


n<^ +1 ; 4)™ = 4"' ( ” ) (?; 3)L ri ^7—~ 4 


'" +I U;4): 


rs-/ 
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_9 ( rn \ _ ( m+l\ , ~ 

The second factor is equal to q \ 2 ' H q (rn) and the third factor equivalent 
to g -2(*>-0( m 2 +1 )( g; q y>-i Next, we j iave 

m b-\-m 

n<« ( l)x2+y2-xi-yi-m+j l— ^ n (?; q)b+j -i ~ # g ( & + 5 + m), 
i=i j =i 


ml mb 

n<« Q)x2—xi+lj t— ^ n« Q ) a+6—C+m+tCj ik* 5 ?)a+6— C+j—1 | (?) ?)a+6— C—Xj—1 

j=l j =1 j=l J=/+l 

~ (?) q) l m.H q (a + b-C + m), 

m l m b 

IK* ; ( l)y 2 -yi-lj ^ n<* ; g) 6+c _ A _ x ,_i n<* 5 ?)b+c— A+j— 1 | (?) Q)b+c—A+Xj+m 

j= 1 J=1 i=i i=i+i 

~ (?) q)m l H q (b + c - A + m). 

By Lemma IA.ll 

H q (m)H q (b + B + m) 

— --- rsj 1 . 

H q (a + b — C + m)H q (b + c — A + m) 

Finally, we have 


m l 

n( v™~* ; 9) ra _ xl _ j+1+Ij ^4 9).+S-CM i+ l+m-J 

i=i i=i 

m b 

xYl(-q C ~ A+l+J ]q)a+b-C+m-l+l-2j JJ ( “? C ~' 4+m+J ; q) a +b-C-m,-x :i -j > 
i=l j=l+ 1 

where the first factor is equivalent to (—g; q) l m and the third factor to (— q\ q) l ~ b - 
If a + b > C + l, the second factor can be expressed as 

m a-\-b—C—l 

\\{~q c ~ A+l+j ] q) a +b-c-i = I] ( ~q c ~ A+l+j \q) m ~ (-?; q)T ~ C ~ l • 

j = 1 i=i 

By a similar computation, this holds also for a + 6 < C + l. In conclusion, (13. lOjl 
is equivalent to VF“ +i_ . Similarly, though with less effort, we find that the first 
determinant in (13.91) is equivalent to W b ~ l , which gives (13. lip . □ 


3.4. Discrete Selberg integrals and Askey-Wilson polynomials. We recall 

some basic facts on Askey-Wilson polynomials [ AW] . Normalizing them to be 
monic (which is not the standard choice in the literature), they are given by 


P n {x) 


P n (x\ a, b, c, d: q) 

( ab, ac, ad] q) n 
2 n a n (abcdq n ~ 1 - 1 q) n 4 


fq n ,abcdq n 1 ,a^,a/^_ 
y ab, ac, ad 



(3.12) 
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where x = (£ + £ 1 )/2. When max(|a|, |6|, |c|, \d\, |g|) < 1, they satisfy the ortho¬ 
gonality relation 


Pm{x)Pn{P) ic(ic) dx C n 8mm 


'-1 


where, using standard notation such as (£ ±2 ; q)oo — (£ 2 ; q)oo(£ 2 ; 0 , ) o o, 

(e ±2 ;g)oc 


w(x) = w(x; a, b , c, d; q ) = 


(a^, &f±, c^ ± , d^\q) ooVl - £ 2 


(3.13a) 


and 


C' n = C n (a, b,c, d; q) 


2n(abcdq‘ 2n 1 ,abcdq 2n ;q) c 


4 n (q n+1 , abq n , acq n , adg n ,6cg n , cdg n , abcdq n ~ l \q)^' 


(3.13b) 


The polynomial P n is symmetric in the parameters a, b, c, d. 

To link Askey-Wilson polynomials to discrete Selberg integrals, we will need 
the Cauchy-Binet identity 


det ( V AjiBik ) 
- hk - rn ) 


1 <j,k< 

as well as the determinant evaluation 


det (AvJ.det {Bi kJ ) (3.14) 

0<Zi<-<Z m <JV 




To prove the latter, we write the determinant as 


(3.15) 


1 —l(&j Q^)m—‘ 


■(aq k ;q) j - 1 (bq k ;q) rn -: 


II ^ (( a ^ fc ; q)j-i(W k -,q)m~j), 


11 (a, 6; <?)j-i i<A<™ 
which can be evaluated using |K11 Lemma 2.2] or [ST, Lemma A. 1]. 

Proposition 3.6. Let 

£q k ~i _|_ £-iqi-k 


x k = 


k — 1,..., m 
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and let P n {x) be the monic Askey-Wilson polynomial (13.121) . Then, 

(lol \<_j.k<rn {P n -\-j —1 


A(x) 


x 


g 2 (™)+( n + 1 )(T) (2a) mTl 

tt {ah, ac, ad] q) n +j-\ 

{ j ( abcdq n ~ l ; g)„ +i _i(g, g 1 ”™-™, of, a^g 1 "™; q) j - 1 


Am) [ Q 

X 403 ' 


1 —m —n 


,abcdq n ,a£,ag 1 4 _ 


Q,Q 


(3.16) 


Proof. Since 

P-ri+j—l {^k) 


ab, ac, ad 

(i ab , ac, ad; q) n + 3 -\ 

{2a) n+ i- l {abcdq n+ j- 2 ] q) n +j -l 

{q 1 ~ j ~ n ,abcdq n+j ~ 2 ,afq k ~ 1 ,af~ 1 q 1 ~ k ]q)i t 

i> o 


(g, ab, ac, ad] q)i 
expanding the left-hand side of (13. 16[) using (j3. 14[) gives 
(ab,ac,ad]q) n+ j i 


n 


L = i {2a) n +i~ 1 {abcdq n+ i~ 2 ] q) n +j -1 


e n 


gh 


0<2i<---<Z m <n+m—1 j 


, ^ (g, ab, ac, ad] q) tj 


x det ((g 1 J n ,abcdq n+] 2 ]q)i k ) det ((a£g J \a£ V J ;gk)- 
Applying (13. 15ft and simplifying, using also 


A(x) = 


g 2 (3)+(?)(2e)(2) ’ 


completes the proof. 


□ 


We note that, since the Askey-Wilson polynomial is symmetric in its parame¬ 
ters, the right-hand side of (13.16ft is invariant under interchanging a and b. This 
proves the following multiple Sears’ transformation, which is a very special case 
of a transformation for discrete elliptic Selberg integrals conjectured by Warnaar 
[Wj and proved by Rains [Raj . We will use this transformation in 1 13.51 


Corollary 3.7 (Rains). If q 1 n abc = def, then 


dm) ( Q 


y l—m—n 


, a, b, c 
4</ ^ 3 l dq m ~ 1 ,e, f ’' 


be 

Q,Q I = ( t 


n 


( b, c] a) j _ 1 (de/bc, df/bc] q) n +j -1 


d J Ji, (d/b,d/c]q)j- i(e,/;g) n+J -_i 

Am) ( q l ~ m ~ n , a, d/b, d/c _ \ 

X 4< ^ 3 ydq m ~ 1 ,de/bc,df /bP^'^J ' 
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3.5. Continuous Selberg integrals. Let // be a linear functional on C[x], which 
we write as a formal integral 

M = f p{x)dfi{x). 

We will assume that // is non-degenerate in the sence that there exist monic 
polynomials p n of degree n such that 

V (PmPn) — C n 8 mn . (3.17) 

We do not require any positivity condition for //. We then have the identity 


/ A(o;) 2 nn«- Xj ) dp(x i) • • ■ dp(x n ) 

■' 3=1 k =1 

n deti<j t k< m {pn+j-l(yk)) fo 1cA 

71 . bo ' ' ' bn—1 a / \ ) fo.ioj 

A(z/) 

relating a Selberg-type integral to a determinant of orthogonal polynomials. This 
identity can be obtained by combining two classical results [I; Thm. 2.1.2 and 
Thm. 2.7.1] due to Heine and Christoffel. More explicitly, it appears in [BHj- A 
direct proof is very easy; simply write the integrand as 

A(x)A (x,y) 

A (y) ’ 

expand both factors in the numerator using A (a;) = det(pj_i(xj)) and then inte¬ 
grate using (13.171) . We can apply (13.181) to prove the following quadratic transfor¬ 
mation formula for determinants of Askey-Wilson polynomials. 

Theorem 3.8. Let m, n, M and N be non-negative integers, with m even, and 
let a, b and q be generic parameters. Let p n and q n denote the monic polynomials 

Pn(x) = P n (x ; aq M , -a, bq N , -6; q), 

'2- n Pp n/2 {2x 2 - 1; -1, —q m+1 , a 2 , b 2 ; q 2 ), 


q n (x) = 


n even, 


2 ( n ^PxP^n-i)/ 2 (2x 2 — 1; — q 2 , —q m+1 , a 2 , b 2 \ q 2 ), n odd 


and let 


Vk — 


Vk + v k 1 


Zk = 


(k + Ci 


-i 


. 7- m+1 

Vk = i q 2 : 


C k = 


aq 


k -1 


bq 


k-M-l 


k = 1 ,pm, 

k — 1,..., M, 

, k = M + 1,..., M + N. 


Then, 


det i<y fc<m (Pn+j —1 i.Vk) ) q deti<j M+N (9n+j—l(^’fc)) 


A (y) 


A (2) 


(3.19) 
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where 


C= (2 


*n 

j =1 \ 


/( g 2 b 72 ]+l 5 _ a 2 q 2lU-l)/2]+^ _ b 2 q 2[U-l)/2]+^ ^2^/2]-!. g 2 ^ 


X 


(a 2 6 2 g 2 -? -3 , a 2 6 2 g 2j_1 ; g 2 ) ™ 

0 a 2 b 2 q 2 i - 3 , a 2 b 2 q 2 ^~ 2 ] q) M +N 


(abqi- 1 , a 2 b 2 qi " 2 ; g)M-Mv( _a V _1 , -abqi~ x ] q) M (~b 2 q j-1 , -abqi~ l ] q) Nj 

Proof. Let L denote the left-hand side of (13.19[) . Since (13.19P is a rational identity, 
we may assume that max(|a|, |6|, |g|) < 1. We then apply (13.18jl to write 

If 


L = 


. ™-i w / a (^) 2 n n^ 

n! llfc=0 kfc “'[- 1 > 1 l n j=lfc=l J=1 


(3.20) 


where w and C*, are obtained by substituting (a, b , c, d ) i->- (ag M , —a, 6^, —6) in 

(EH- 

We will now rewrite (13.201) in such a way that the roles of m and M + N are 
interchanged. In the orthogonality measure, we write 

(£ ±2 ;<?) co 


_ = « ; , gM£ ±4 ; g 2 )oc 

(ag M £±, -af± — &C= t ; g)oo (~£ ±2 , ~qf ±2 , a 2 f ±2 , b 2 f ±2 ] q 2 ) 

and observe that 

M+N 

K±; ?)*«*;«)» = 2" +K 9 (?) + ffla*W r (2t - x). 


fc=l 


We also write 


II (y k - x ) 


fc=l 




{-q^q 2 ) 


= 2~ m q 


{-q m + 1 ^ 2 -q 2 ) c 


Combining these facts, we find that 


L = 


D 


n M+N 


where 


and 


n! Hi 


w(x) = 


j— f a{x) 2 n n (^ - Xj) jj 

0 C k ■/[-l.l]" j=\ k=l 3=1 


w(xj) dxj , 


(3.21) 


3 =L /e=l 

(£ ±4 ; r)c 


(-£ ±2 , —g m+1 £ ±2 , a 2 £ ±2 , b 2 f ±2 -, g 2 )oo\/l - a; 2 


D = (2 M+N - m q(^) + ^)-^a M b 
We now apply ([3J~8j) to the integral (13.211) . Let 


p(p) — I p ( x ) h)'(x) dx. 
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Then, 


w(x) dx = - w{y\ -1, ~q m+1 , a 2 , b 2 ; g 2 ) dy, 


1 


® w(z) cte = - w(y, -q , -g m+ ,a ,b]q)dy, 

where y = 2x 2 — 1. It follows that the polynomials g n (+) satisfy y(q m q n ) = C n S mn , 

with 




4 k Ck(— 1, — g m+1 , a 2 , 6 2 ; g 2 ), n = 2k, 

4 _fc_1 Cfc(—g 2 , — g m+1 , a 2 , fr 2 ; g 2 ), n = 2k + 1. 


Thus, (13.18P gives 


L = 




Cj _ 1 det 1 k< M+N ( gn+ J - 1 (■) ) 


, =1 C't-i A(z) 


Simplifying the expression for Cj-i/Cj-i, we arrive at the desired result. 


□ 


We will now combine (13.16(1 and (13.191) . Let us substitute (a,b,c,d,£) t->- 
(iag M , — iot,i/3q N , —i/3,iq ( - 1 ~ m ^ 2 ) in (|3.16j) . where M and N are non-negative inte¬ 
gers. Then, the left-hand side of (13.16jl equals the left-hand side of (13.191) . under 
the substitutions (a, b) (io;, i/3). We rewrite the matrix entries q n+ j_i(zk) in 
terms of 4</> 3 series, using (13.121) with the distinguished parameter a chosen as —a 2 
for k < M and as — /3 2 for k > M + 1. Then, the matrix entries in (13.191) can be 
identified with those in (12.71) . More precisely, 


m +1 


q n +j-i(zk) = C j D k Q“" n (a,l3,q-i-- t q), 


where 


Ci = 


(i/ 2 ) 


m+1 T n+j — 11 

n+j-l q 2 [ 2 J 


(n+j- 2) 2 
4 


J (a 2 /3 2 ;g-) [(n+j i )/2] 


(a 2 /3 2 g' m + 2 [(n+i)/ 2 ]- 1 ;g 2 ) [(n+ ._ 1)/2] 
(a / d) fc 1 g( fc 2 1 ) 


Du — < 


(a 2 /3 2 ; g 2 )fc-i ’ 


(otP) 


k-M-l, 


k (a 2 /3 2 ; q 2 )k-M-i 


k < M, 
k>M+ 1. 


We compute 


A(,) = 


Ylk=i(<h-q j la - 2 ',q)j-i(q j M Pfa,-q j 1 ®P\q)N]A j=1 (q,-q j 1 P 2 ]q)j -1 


2 (T)i(T)g 2 (I)+ 2 (^C)dM + i)(f) a (:)^)w 

and simplify the factors involving a/3 using 

1 

nil <?M« 2 /3 2 ; q 2 )^ f[L (« 2 /? 2 ; gVi 
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tt _ (a 2 /3 2 -, g 2 )[ (n+ j-1)/2] _ tt (a 2 f3 2 g 2j 3 ,a 2 f3 2 q 2j 2 -,g) M +N 

AA (a 2 /3 2 g TO + 2 [( n +j)/ 2 ]- 1 ;g 2 ) [(n+i _ 1)/2] (a 2 /^^'- 1 , a 2 /3 2 g 2 J“ 3 ; g 2 )m 

A__ 

X J = i (-a/9^- 1 , a 2 /3 2 q->- 2 -, q) M +N(a/3q^~ 1 - 1 q) M {a/3q j ~ 1 -, q) N 

El. f=i(oiPq n+3 ~ 1 -,q)N 

U-L^i^^q 2 '^- 2 ; q)j -1 

^ (a 2 /3 2 q 2j ~ 3 ) q 2 )^ M+N+ n + i j ( a 2 (3 2 q 2j ~ l ; g 2 ) jM+^+nj 
X ij (a 2 /3 2 q 2 i- 3 ; q 2 ) M +N+n(a 2 /3 2 q 2j - 1 ) q 2 ) n 


In the exponent of q, we use 


A' — 1 |"n + j — 1] I"(n + j — 2) 2 
£ (—0 - $— + 4 


1 ffM + N + n 

2 V V 3 


m f I" (M + N + n — l) 2 
T V 4 


(n — l) 2 
4 


This yields the following result. 

Corollary 3.9. For m, n, M and N non-negative integers, with m even, 


Am) fq 1 m n ,a 2 /3 2 q M+N+n 1 ,aq M ™ + 2 ,—aq M ™ + 2 \ 

4</>3 V a 2 q M ,a(5q M ,-a(5q M+N ’ q ’ q ) 

— (_l)( M 2 iY )+ n ( M + iV +T) Q /( M + m ) n + 2 (^) + (^) / g( M + n ) iV +( 2 f )+ 2 ( 2 r )g X 

TT_ (a/?g :,+Tt ~ 1 ; g)jy _rr_1_ 

A = 1 (g, -aV'- 1 ; q)j-i(q j - M f3/cr, q) N ^ (g, AVA ?)j-i 

tt (aW J '~ 3 ; g 2 )[(M+jv+»+i)/2](q 2 /3 2 g 2i ~ 1 ; g 2 )[( M +jv+n)/2] 

X (a 2 /3 2 g 2j_3 ; g 2 )M+JV+n(a 2 £ V i_1 ; g 2 ) n 

A (a 2 /3 2 g M+Ar+n_1 ; g) n+i _i(g, g) j _ 1 (a 2 g 2M - m+1 ; g 2 ),,_i 


(a 2 g M , af3q M , - a/3q M+N ; g) n+i _i 

(g 2 ^/2]+l, a 2 g2[( J --l)/ 2 ]+l > /3 2 ? 2[0--l)/2]+l. M4W 

(a 2 gJ- 1 ;g) A f(/3y- 1 ;g)jv i. 


jg A TT 1 

( a 2p2 q j+2n-2. ? ) j _ 1 


x Q MNn (a, f3, q 1 ? q), 
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where 
X = 2 


+ 

M 


m 


+ 3 


M 


+ 3 


+ 


M 

2 


+ (M + 1) 


+ 


, , , m 1 

+ m [ M H- 

' 4 2 


1 

n -\— 
2 


N 

M + N + n 
3 


+ 


m 


(M + N + n- l ) 2 


(n — l) 2 


We now reformulate Corollary 13.91 in a way that will be adapted to our purpose. 
We first apply the identity 


M 

n 


L = i g)j~ 1(aV _1 ; g) 


n 


2_2[(j-l)/2]+l. „2 




;g 


M 




X 


M 

n 


" ( ? - 2 [(,--l)/ 2 ]+l-m /a 2 ;? 2 )ia 

11 


^ (-q j+1 ~ 2M /a 2 ; q)j -r ^ g ) M 

We then multiply the left-hand side by 

(a/3q M , -af3q M+N ; g) n +j-i 


~±mnM 


n 

j=i 


( a 2 g 2M- m + 1 ;g 2)._ l 


and make the change of variables 

(M,N,a,P) ^ (iMllNlaq-^^sgniMN)^-^ , 

where we no longer require M and N to be non-negative. Thus, for M > 0 we 
consider 


-3-mnM 


n 

3 = 1 


(a/3q M l N 


-apq 2 -q) n+ j x 


( rv 2 n M-m-\-l. n 2\ 

[a q , q )j -1 


x 40 : 


(m) I q 1 m n ,a 2 /3 2 q n 1 ,aqi 

a 2 , a/3q^ + % , — af3q^ 


1 M 

2, —ag 2 ' 

M__N_ 

2 


;g,g (3-22) 


and for M < 0 the same quantity with (M, 0) i—)■ (-M, —f3). However, by Corol¬ 
lary 13.71 (I3.22jl is invariant under this transformation. Thus, we may take the 
left-hand side as (I3.22p regardless of the sign of M. This leads to the following 
result. 


Corollary 3.10. For m, n, M aiid N integers, with m and n non-negative and 
m even, and £ = sgn(MiV), 
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i4>t ] 1 q 


1 —m—n 9 ol n — 1 iH_2Z!xi 

.apq \aq 2 2+2 ,—cvq 2 2+2 

2 m+al M_iv ;g,g 

cr, apq 2 + 2 , —apq 2 2 

_l)( l 2 l ) £ ('" l )+ iV ( rl + M )Q ! ( 2m -| M l) n +( l 2 l ) / gHA r l + ( |M| J |JV| ) + ( l 2 l )g^ 

|M| 


M m 1 1 


x 


x 


X 


X 


-ea/3q j+n 


, \M\ [jVJ 

2 2 




i=i (g,-g i+1_|M| /Q! 2 ;g)j-i(-£g j '"'p/ar, q)\ N \ 

(o; 2 / S 2 g 2 - 7 ' _3_ l M l _ l Ar l; q’ 2 ) pM|+|iv|+»+i 1 (a 2 ^ 2 q' 2j '~ 1 ~ |A/| ~ |JV| ; g 2 ) r iMj+jjvh^ 
i=i 


n 

3= 1 
n 

n 

i=i 

|M +\N\ 


{a*PW j -*~ l M| -l JV l; 9 2 )|M|+|Ar|+n(a 2 /3 2 g 2 - 7_1_ l M l-l Jv l; g 2 ), 
(a 2 /3 2 g n_1 ; q) n+j -i(q, q g) i _i(a 2 g Af - m+1 ; g 2 )j_i 

i\/ + A r 


(a 2 ,a/3q 2 ,-afiq 2 ■ 1 q) n+j _ l 
( q 2\j/2]+l^ q -2[(j-l)/2]+l-m+\M\/ ^2, ^2 ? -|JV|+2[(j-l)/2]+l. g 2 


( g i+ 1 - n /a 2 ;g)|M|(^ 2 g- 


2„j-l-|JV|. 


; ?)mvi 


X 


- ^ (a 2 /3 2 g^'+ 2n_2_ l M H Ar l; g)j_i J-Jj- (g, — gi—1^1—!/5 2 ; g)j_i 


7V| 

n 


X g|M|,|AT|,n( a g 1 2 1 5 1 2 1 , g^a" 1 ), 


where 


X = 2 


m\ l m 
3 1 + 


-:.Y|(|.\/| 2 — |.V/.V| - 21.V| -2) 


+ -{m(m + M — 1) + |M|(|M| + 1 — m — n) — |iV|j 


n 


1 

2 

m 

~2 


\M\ + \N\ + 

(|M| + |iV| + n — l) 2 ’ 


+ 


[n — l) 5 


3.6. Final steps. We can now complete the proof of Theorem 12.11 Assume first 
that m is even. Combining Corollary 13.21 Proposition 13.41 and Corollary 13.101 with 
the substitutions 


(a,P,n) ^ +a+c+m \b), 


yields f lATUl) . with 


c = 


^ 2 ^m{a-\-b+M+N)-\-ab 


n 

3= 1 


(g, g)a+j-i(g, o)c+m+j— 1 
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X 


X 


n 


(s; q)j-i(q 2 ;q 2 )i(a+c+N) +j -i 


L = -L (g; q) a+t+M+N +J -_ 1 (g 2 ; g 2 ) t+c-M +j _ 1 

eq 


\M\ 

n 


■ I o , b \M\ |JV| 

-P^+2 + 2 2 2 J g)|Af| 


=1 ( q , — qj+b+c-\M\+m., £gJ + 2 + 2 


#+ 4 +c+m _M_M 


; ?)ijvi 


,n/2 (g2j — 1+a—6—|M|—|7V|. q 2 ) r |M|+|JV|+i , +1 1 ( q 2 j+l+a-6-|M|-|jV|. ^,2^ 


X 


n 

j =1 


|M| + |A| + b j 


(g2 j-l+a-6-|M|-|AT|. g 2 )|M|+|Ar|+ft(g 2 i+1+ a-6-|M|-|iV| ; g 2 )fe 


^ ^g 2 b7 2 ]+l ) g~ 2 [(i — l)/2]+6+c+|M|^ ^q+e+m—|A r |+2[(j —1)/2]+2. q^\rn 

H ( gJ + C +m ;g )| M |( g i + a + c +m -|7V| ;g )| 7V | 


| M 

+\N\ 

N \ , 


IT 1 1 

n 1 

2 

LI (g2+a+b-|M|-|AT|; q'jj ! ■ 

L = -L (q, -gt+a+c+m-liVI^)^! 


for a certain exponent A". 

We now express C in terms of the function H q . The product YYj=i is equal to 


m/2 

n 


(q-,q 2 )[(a+i)/2]+j-i(q-,q 2 )[a/2]+j 


L = 1 ( 9 ; q 2 ){a+b+\M\+\N\)/2+j-l(q] q 2 )(a+b-\m\-\N\)/2+j 

f g+l l , m—1 [a] , m+1 a+b+|M| + |JV|-l q+b-|M|-|iV|+l \ 

L 2 J ' 2 ’ L 2 J 2 ’ 2 ■ 2 / 


H q 2 


TT ( Ta+ll 1 I" a~\ I 1 a-\-b-\-\ M | + | N\-\-m— 1 a-\-b— \M\ — |iV|+m+l\ 

- n 9 2 \[ 2 J 2’ l_ 2 J _l " 2’ 2 ’ 2 y 


(3.23) 


All other factors can be converted using Lemma IA.4I and Lemma IA.5I The end 
result can then be simplified using Lemma IA.31 For instance, the factor H q (a) 
appearing from 17^=1 (#; 9)a+j-i combines with factors from (13.231) as 


H q {a) 


Mffl-llil + i) 

We also observe that 


= H q2 


( 

'a' 


CL ~\~ 1 

l 

L 2 J 


2 


\M\ 

11 y ‘ 


JT 2 


H: 


; q)\m = 


3 = 1 


H: 


- £ ( a+b—\M\ — \N\ q+fe+|M| + |jV| \ 

2 ’ 2 J 

_ £ ( a+b+|M|-|iV| a+b-|M| + |iV| \ 


H q 2 


a+b—|M| —|W| a+b+|M| + |JV|\ 

2 ’ 2 J 


tj f a+b—M—N a+b+M+N 
- n <7 V 2 ’ 2 


)«,-( 


- ( a+b+M-N a+b-M+N\ ’ 


2 


2 


2 


2 
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where most factors on the right eventually cancel. Applying Lemma [A.51 to 


Yl(-q^- a - b+M ~ N)+j ;q)a 

3 =1 


leads, apart from functions H q and a power of q, to the factor 

JJ _ D( a _ b+M _ N y 2 D(_ a+b+ M-N)/2 
D(-a-b+M-N)/2D(a+b+M-N)/2 ’ 

where D k = 2 mm ( fc ’°). Since —a — b + M — N < 0 and a + b + M — N > 0, it is easy 
to see that D = 2^~^ max (l a_b l’l M_iV l). Finally, we express H q in terms of H q . In 
this way, we find that (12. 101) holds up to some factor q x , where X is independent 
of q. Since all functions in (12. 10)1 are invariant up to a sign under replacing q by 
g _1 (here we use (12. 9H ), we must have X = 0. This proves Theorem 12.11 in the 
case when m is even. 

If m is odd we invoke Lemma 13.51 Since a rational function is determined by 
infinitely many values, it is enough to prove that the right-hand side of ( 12 . 10 p . 
considered as a function of m, has the same form as the right-hand side of (13.71) . 
Since (I2.7j) is a Laurent polynomial in a, /?, 7 , the second factor in ( 12 . 101 ) is rational 
in cf 1 ! 2 . The final factor in ( 12 . lip , involving Hf £ , is rational in q m by Lemma [A.31 
Consider now the remaining g-hyperfactorials in (12. lip . We rewrite H~ as H q 2 /H q 
and then apply Lemma [A.31 to all factors of the form f(x + m), where / = H q or 
H q 2 . This leads to a product of the form 


6 


n 


H* (% + f) TT 

^ fe+ f ) T 


('i + V) TT (ej + f) 
H,, (dj + =yi) T H,, (/, + f) 


TT H,, fa + 

l_i H,. (hi + !yi) • 


where a±,, h\o are all integers. One may check that 


6 10 10 10 

bj ~ a o ) = — c j ) = — fj) = y 'S -h ~ hj) 

j = 1 3 =1 3 =1 3 =1 

a + b + M + N 

= ---= a + b - C. 


Applying Lemma [A.21 it follows that the right-hand side of (12 . 10p indeed behaves 
as (13. lip as a function of m. This completes the proof of Theorem 12.11 


Appendix. The c/-hyperfactorial 

In this Appendix, we collect some elementary properties of the q- hyper factorials 
H q and H q defined in (12.61) . 

Lemma A.l. Let Gp,... , a n , b \,..., b n be non-negative integers, such that ^2 a 3 = 
J2bj an d let H be any one of the functions H q , H~, H q or H~. Then, there exists 
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a rational function f such that 


n^±|). /( ^) 


a h ( b - +f) 

for each non-negative integer m. 

Proof. It is easy to check that 
m\ 


(A.l) 


H q [a + —J 


m even, 


H q {m/2)(q ; g)^ /2 n./=i(g 1+m/2 ; g)i-i, 

^ g ((m +i)/2)(g i /2. g )“ m+1)/2 n ; =1 (q,l+m/ 2 . m odd. 

It follows that (1A.1I) holds for H = H q , with 


(A.2) 


f( x ) = 11 


1 =1 n.'y fc =i(^; Q)j -1 

Let us now replace H q by H q . If m is even, (IA.1D is multiplied with gQl"*)/ 2 , 
where Q is the polynomial 


<2M = J] 


k =1 


b k + ? + 1 


Ofc + y + 1 


Note that the cubic and quadratic terms in Q cancel, so qQ( m )/ 2 is a rational 
function in q m / 2 . Moreover, since §( 2 ™ +1 ) = ( m ^ 1 ) + the same multiplier 
appears when m is odd. This proves the case H = H q . The remaining two cases 
follow since Hf = H q 2 /H q and H~ = H q 2 /H q . □ 

We will also need the following variation of Lemma IA.1I 

Lemma A.2. Let a q , bj, Cj and dj be non-negative integers such that 

k i 

~ 6 i) = X^ ~ = X - 

3 =1 3 =1 

Then there exists a rational function f such that 

rf g h a i + f) rT H A c i + m f L ) _ i/2.„i/2n V 

/or eac/i non-negative integer m. 

Proof. It follows from (1A.2I) that 

* H, (aj + f) J (r^ i ’ n+2)m (i;q) m , 2 y f(q m/2 ), 


n 


m even, 


= i #<? + ' 2 ) 1 (q i6( m+1)2 (g 1/2 ;g)( m+1)/2 ) /(g m/2 ), m odd, 
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where / is rational. Replacing m by m — 1, it follows that 

A 

H 0 (Ci + s=±) 


-J-m 2 

q i6 


(q 1/2 'i q) 


m even 


tt y-j ~-) _ ) v ^ ’i'™/ 2 ) g(q m/2 )^ 

=i faj d 2 ) 1 g)( m _i)/ 2 j g(q m ^ 2 ), m odd, 

again with g rational. Using 

/ 1 / 2 . i/ 2 \ _ \(q 1/2 ',q)m/ 2 (q-,q)m/ 2 , m even, 

\(^ 1/2 ;?)(m+i)/ 2 (g;g)(m-i)/ 2 , m odd 

we obtain the desired result. 

The following two lemmas are straight-forward to verify. 

Lemma A.3. For m a non-negative integer, 

1 m m m + 1 


H q (m) = H q 


m 


2 7 2 1 2 1 

The same identity holds if H q is replaced by H q . 
Lemma A.4. For k, l and m non-negative integers, 

Hq{k + m) 


q)m+j -1 

3 = 1 
k 

I ~[(q l+3 -,q)m 

3 = 1 
k 

\[q'''-q).i ' 

t=l 


H q (l, k + l + m) 

H q (k + 1,1 + m ) ’ 

Uhf U ¥ + *=4 + *) 


ff,(i +1) 

Moerover, for k, 2/ + 1 and m non-negative integers, 

H q {l, l, l + m + [fc/2], l + ?77. + \{k + 1)/2]) 


H(q l+1+[ij ~ 1)/2l ,q) m = 

3 =1 

n(<z ,+1+b72] ;<z)r, 

i=i 


□ 


H q {l + 771, l + 771, l + [k/2], l + \fk + l)/2]) 

H q (l, l + 1,1 + 7Ti + [( k + l)/2], l + 77i + [(k + 2)/2]) 

Hq{l + 771, l + 771 + 1, l + [(fc + 1)/2], l + [(fc + 2)/2]) 


TT , I | k j o- 1 j F[ q (l — £,l + £,l + 771 + k/2, l + 171 + k/2 ) 

Tim H q (l + m — £,l + m + £,l + k/2,l + k/2)' 


3 = 1 


where £ = 0 /or k even and £ = 1/2 /or k odd. 

We also need the following variation of the second identity in Lemma IA.41 
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Lemma A.5. For k and m non-negative integers and l an arbitrary integer, 


k 

n< 

3=1 


-q l+J \q)m = 


Ck+iCi +m H q (\l\,\k + l + m\) 


CiCk+i+m H q (\k + l\, \l + m\) ’ 

where C n — 1 for n > 0 and C n = 2 n q( n ~ n3 )/ e f or n < 0. 

Proof. By induction on k, the result is reduced to 


{-q k+l+1 - 1 q) m = 


Ck+i+iCk+i+ m H (\k + l + m+l\,\k + l\) 


C < fc+/Cfc+i+m+1 H q (|/c -|- l T 'nrI, \k T l T1|) 

By induction on m, this is in turn reduced to 

H~(\k + l + m\,\k + l + m + 2|) 


1 + q 


k+l+m+l 


n 2 

^ k+l+m+l 
C' k+l+m,Ck+l+m+2 


H~(\k + l + m + l\) 2 


It is easy to check that 



H q (|n + 1[, \n- 1|) 

H q {\n\) 2 

Thus, the result holds for any solution to the recursion 

1, n > 1, 

= {2, n — 0, 




C n -\C n+ i 

The given solution corresponds to the initial values Co — C\ — 1. 


q n , n < —1. 


□ 
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